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William  Fast — COL  Fast,  USA  (Ret.),  facilitates  acquisition  and  program  management  courses  at  the 
Naval  Postgraduate  School.  He  also  writes  and  speaks  on  various  management  topics  and  provides 
consultation  services  to  defense  acquisition  programs.  From  2006-2010,  COL  Fast  taught  program 
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Abstract 

The  2013  Annual  Report  on  the  Performance  of  the  Defense  Acquisition  System 
(USD[AT&L],  June  28,  2013)  identified  a  need  for  additional  analysis  of  recent  acquisition 
policies  (DoDI  5000.02,  December  2008)  to  reduce  program  technical  risk  by  requiring 
competitive  prototyping  for  risk  reduction  and  a  Preliminary  Design  Review  (PDR)  before 
Milestone  B.  This  research  uses  cost  and  schedule  estimates  from  Major  Defense  Acquisition 
Programs  (MDAPs)  to  determine  if  competitive  prototyping  and  PDRs  held  prior  to  Milestone 
B  have  resulted  in  mature  technologies  (defined  as  Technology  Readiness  Level  6  or  above) 
and  subsequently  reduced  program  cost  and  schedule  slippage. 

Introduction 

Research  Purpose  and  Questions 

The  purpose  of  this  research  is  to  determine  if  the  DoD  Instruction  5000.02  policies 
for  Major  Defense  Acquisition  Programs  (MDAPs)  relating  to  competitive  prototyping, 
technology  readiness,  and  Preliminary  Design  Review  (PDR)  prior  to  Milestone  (MS)  B  are 
having  the  desired  effect  on  program  outcomes.  To  help  evaluate  these  policies,  the 
following  research  questions  are  answered: 

1 .  Does  the  knowledge  from  competitive  prototyping  and  a  PDR  conducted  prior 
to  MS  B  result  in  better  decisions  relative  to  requirements,  design,  and 
resources? 

2.  What  are  the  effects  of  the  competitive  prototyping,  technology  readiness, 
and  PDR  policies  on  program  costs  and  program  schedules? 


Background 

On  June  28,  2013,  the  Under  Secretary  of  Defense,  Acquisition,  Technology,  and 
Logistics  (USD[AT&L])  released  his  Performance  of  the  Defense  Acquisition  System  2013 
Annual  Report.  In  the  preface  to  the  report,  The  Honorable  Frank  Kendall  called  for  data- 
driven  analysis  to  find  out  what  was  working  in  the  defense  acquisition  process.  The  report 
concludes  with  several  policies  and  processes  for  additional  analysis,  including 

•  Setting  Preliminary  Design  Review  (PDR)  before  Milestone  B 

•  Competitive  Prototyping  for  Risk  Reduction,  not  Proof  of  Concept 

•  Reducing  technical  risks  during  Technology  Development  phase 

(USD[AT&L],  2013b,  p.  105) 

These  three  policies  all  relate  to  technical  risk  reduction  through  competitive 
prototyping  to  inform  requirements  trades  prior  to  Milestone  B.  These  policies  were  first 
included  in  the  December  8,  2008  version  of  DoD  Instruction  5000.02,  Operation  of  the 
Defense  Acquisition  System.  The  next  year,  several  of  these  policies  were  codified  in  the 
Weapon  System  Acquisition  Reform  Act  (WSARA)  of  2009  (Public  Law  1 11-23;  WSARA  of 
2009,  §§  201  and  203). 
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However,  the  policies  actually  begin  with  a  memorandum  by  a  former  USD(AT&L), 
the  Honorable  John  Young.  On  September  19,  2007,  Young  mandated  that  the  Military 
Services  and  Defense  Agencies  “formulate  all  pending  and  future  programs  with  acquisition 
strategies  and  funding  that  provide  for  two  or  more  competing  teams  producing  prototypes 
through  Milestone  (MS)  B.”  Young  went  on  to  claim  that  “competing  teams  producing 
prototypes  of  key  system  elements  will  reduce  technical  risk,  validate  designs,  validate  cost 
estimates,  evaluate  manufacturing  processes,  and  refine  requirements.  In  total,  this 
approach  will  also  reduce  time  to  fielding”  (USD[AT&L],  2007).  Young’s  policy  memorandum 
and  related  policies  on  technology  readiness  and  the  use  of  an  early  PDR  were  incorporated 
into  the  December  8,  2008,  version  of  DoD  Instruction  5000.02,  Operation  of  the  Defense 
Acquisition  System,  as  follows  in  Table  1. 

Table  1.  Policies  as  Found  in  DoD  Instruction  5000.02,  December  8,  2008 

(DoD,  2008,  pp.  17-21) 


Reference 

Policy 

Competitive 

Prototyping 

Enclosure  2 
Para  5c(9) 

The  Technology  Development  Strategy  (TDS)  and  associated  funding  shall  provide  for 
two  or  more  competing  teams  producing  prototypes  of  the  system  and/or  key 
system  elements  prior  to,  or  through,  Milestone  B.  Prototype  systems  or  appropriate 
component-level  prototyping  shall  be  employed  to  reduce  technical  risk,  validate 
designs  and  cost  estimates,  evaluate  manufacturing  processes,  and  refine 
requirements.  Information  technology  initiatives  shall  prototype  subsets  of  overall 
functionality  using  one  or  more  teams,  with  the  intention  of  reducing  enterprise 
architecture  risks,  prioritizing  functionality,  and  facilitating  process  redesign. 

Preliminary 

Design 

Review 

Enclosure  2 
Para  5d(6) 

When  consistent  with  Technology  Development  Phase  objectives,  associated 
prototyping  activity,  and  the  MDA-approved  TDS,  the  PM  shall  plan  a  Preliminary 
Design  Review  (PDR)  before  Milestone  B.  PDR  planning  shall  be  reflected  in  the 

TDS  and  shall  be  conducted  for  the  candidate  design(s)  to  establish  the  allocated 
baseline  (hardware,  software,  human/support  systems)  and  underlying 
architectures  and  to  define  a  high-confidence  design.  Ail  system  elements 
(hardware  and  software)  shall  be  at  a  level  of  maturity  commensurate  with  the  PDR 
entrance  and  exit  criteria.  A  successful  PDR  will  inform  requirements  trades; 
improve  cost  estimation;  and  identify  remaining  design,  integration,  and 
manufacturing  risks.  The  PDR  shall  be  conducted  at  the  system  level  and  include 
user  representatives  and  associated  certification  authorities.  The  PDR  Report  shall  be 
provided  to  the  MDA  at  Milestone  B  and  include  recommended  requirements  trades 
based  upon  an  assessment  of  cost,  schedule,  and  performance  risk. 

Technology 

Readiness 

Enclosure  2 
Para  6c(4) 

Final  Requests  for  Proposals  (RFPs)  for  the  EMD  Phase,  or  any  succeeding  acquisition 
phase,  shall  not  be  released,  nor  shall  any  action  be  taken  that  would  commit  the 
program  to  a  particular  contracting  strategy,  until  the  MDA  has  approved  the  Acquisition 
Strategy  The  PM  shall  include  language  in  the  RFP  advising  offerors  that  (1)  the 
government  will  not  award  a  contract  to  an  offeror  whose  proposal  is  based  on  CTEs 
that  have  not  been  demonstrated  In  a  relevant  environment,  and  (2)  that  offerors 
will  be  required  to  specify  the  technology  readiness  level  of  the  CTEs  on  which  their 
proposal  is  based  and  to  provide  reports  documenting  how  those  CTEs  have  been 
demonstrated  in  a  relevant  environment. 

These  policies  were  further  refined  as  a  result  of  the  WSARA  of  2009.  On  December 
4,  2009,  The  Honorable  Dr.  Ashton  Carter,  USD(AT&L)  issued  a  Directive-Type 
Memorandum  (DTM)  09-027 — Implementation  of  the  Weapon  Systems  Acquisition  Reform 
Act  that  amended  DoD  Instruction  5000.02  (December  2,  2008).  Specifically  related  to  the 
competitive  prototyping  policy,  DTM  09-027  directed, 

The  technology  development  strategy  (TDS)  for  each  MDAP  shall  provide  for 
prototypes  of  the  system  or,  if  a  system  prototype  is  not  feasible,  for 
prototypes  of  critical  subsystems  before  Milestone  (MS)  B  approval. 
Information  technology  initiatives  shall  prototype  subsets  of  overall 
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functionality,  with  the  intention  of  reducing  enterprise  architecture  risks, 
prioritizing  functionality,  and  facilitating  process  re-design. 

In  addition,  DTM  09-027  provided  for  waivers  to  the  competitive  prototyping  policy: 

a.  The  MDA  may  waive  this  requirement  if:  (1 )  The  cost  of  producing 
competitive  prototypes  exceeds  the  expected  life-cycle  benefits  (in  constant 
dollars)  of  producing  the  prototypes,  including  the  benefits  of  improved 
performance  and  increased  technological  and  design  maturity  that  may  be 
achieved  through  competitive  prototyping;  or  (2)  The  Department  of  Defense 
would  be  unable  to  meet  critical  national  security  objectives  without  such  a 
waiver. 

b.  If  the  competitive  prototyping  requirement  is  waived,  the  MDA  shall  require 
the  program  to  produce  a  prototype  before  MS  B  approval  if  the  expected  life- 
cycle  benefits  (in  constant  dollars)  of  producing  the  prototype  exceed  the  cost 
of  the  prototype  and  the  production  of  the  prototype  remains  consistent  with 
national  security  objectives  (e.g.,  when  the  initial  operational  capability  must 
be  fielded). 

(DoD,  2009,  Attachment  1 ,  pp.  4-5) 

Regarding  the  PDR  policy,  DTM  09-027  directed  that  “PDRs  before  MS  B  are 
mandatory  for  all  MDAPs  and  will  be  reflected  in  the  TDS  to  be  approved  by  the  MDA  at  MS 
A.  Post-PDR  assessments  will  be  conducted  in  association  with  MS  B  preparations  and  will 
be  formally  considered  by  the  MDA  at  the  MS  B  certification  review”  (DoD,  2009,  Attachment 
1,p.  9). 

So,  in  addition  to  providing  for  prototype  waivers,  DTM  09-027  changed  competitive 
prototyping  and  the  requirement  for  a  PDR  before  MS  B  from  policy  to  a  statutory  mandate 
for  all  MDAPs. 

Finally,  DTM  09-07  provided  a  template  for  the  Section  2366b  of  Title  10  certification 
required  from  the  Milestone  Decision  Authority  (MDA)  to  the  Congress.  By  signing  the 
certification  memorandum,  the  MDA  attests  that  various  statutory  requirements  have  been 
met  by  the  program  before  entering  the  EMD  phase.  Among  the  sample  language  provided 
in  DTM  09-027  are  these  statements  related  to  the  PDR  and  technology  readiness  (see 
Appendix  A  for  TRL  definitions): 

I  have  received  the  results  of  the  preliminary  design  review  and  conducted  a 
formal  post-preliminary  design  review  assessment,  and  certify  on  the  basis  of 
such  assessment  that  the  program  demonstrates  a  high  likelihood  of 
accomplishing  its  intended  mission. 

The  technology  in  the  program  has  been  demonstrated  in  a  relevant 
environment,  as  determined  by  the  Milestone  Decision  Authority  on  the  basis 
of  an  independent  review  and  assessment  by  the  Director  of  Defense 
Research  and  Engineering. 

(DoD,  2009,  Attachment  3,  pp.  16-17) 

On  November  26,  2013,  The  Honorable  Frank  Kendall,  USD[AT&L]  canceled  DTM 
09-027  after  incorporating  all  of  the  statutory  requirements  of  the  WSARA  of  2009  into 
Interim  DoD  Instruction  5000.02,  Operation  of  the  Defense  Acquisition  System. 

It  should  be  noted  that  20  years  before  The  Honorable  John  Young,  USD[AT&L] 
mandated  competitive  prototyping  for  MDAPs,  essentially  the  same  requirement  appeared 
in  the  September  1987  version  of  DoD  Instruction  5000.2.  That  earlier  mandate  was  termed 
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Competitive  Prototyping  Strategy  (CPS)  and  was  required  for  all  MDAPs,  if  practicable 
(DoDI  5000.2,  1987).  One  impetus  behind  the  mandate  was  the  Packard  Commission’s 
recommendation  to  place  “high  priority  on  building  and  testing  prototype  systems  to 
demonstrate  that  new  technology  can  substantially  improve  military  capability,  and  to 
provide  a  basis  for  realistic  cost  estimates  prior  to  a  full-scale  development  decision” 
(Packard,  1986,  p.  55).  If  a  CPS  was  not  planned,  the  Congress  had  to  be  notified,  in 
accordance  with  Section  2365,  Title  10,  United  States  Code  (DoDI  5000.02,  1987,  p.  7, 
paragraph  F4).  This  CPS  requirement  lasted  about  five  and  a  half  years  before  it  was 
canceled  by  Change  1  (February  26,  1993)  to  DoD  Instruction  5000.2  (February  23,  1991). 
This  was  just  a  few  months  after  Public  Law  102-484  (October  1992)  repealed  the  required 
use  of  a  competitive  prototyping  strategy  for  MDAPS.  The  change  appears  to  have  been 
part  of  the  wave  of  acquisition  reforms  that  reflected  the  need  to  reduce  post-Cold  War 
defense  spending. 

Literature  Review 

An  important  work  on  prototyping  in  DoD  weapon  system  development  was 
published  in  1992  by  RAND.  The  author,  Jeffery  A.  Drezner,  constructed  a  database  of 
programmatic  information  for  287  programs  and  surveyed  43  government  program 
managers  about  their  use  of  prototypes,  from  about  1960  through  the  late  1980s.  Fie 
categorized  how  prototypes  were  used  in  support  of  various  weapon  system  acquisition 
strategies.  Fie  specifically  quantified  cost  growth  and  schedule  slippage  in  aerospace 
programs  from  the  Selected  Acquisition  Report  (SAR)  of  December  1988.  Drezner 
concluded  that  the  relationships  between  prototyping  and  cost,  schedule,  and  performance 
outcomes  were  “ambiguous  due  to  the  effect  of  confounding  variables”  (Drezner,  1992,  p. 
68).  Fie  could  not  find  any  significant  statistical  differences  between  outcomes  from 
prototyping  versus  nonprototyping  programs.  Drezner  did  not  recommend  that  DoD  change 
its  DoDI  5000.2  (February  23,  1991)  policy  mandating  competitive  prototyping  for  all  major 
weapon  systems.  Rather,  he  recommended  that  each  program  manager  weigh  the  costs 
and  benefits  of  prototyping  against  the  risks  and  consequences  of  proceeding  into  the  next 
phase  without  the  knowledge  gained  from  prototypes  (Drezner,  1992,  pp.  59-67). 

A  more  recent  RAND  study  includes  a  chapter  entitled  “On  Prototyping:  Lessons 
from  RAND  Research.”  The  authors,  Jeffery  A.  Drezner  and  Meilinda  Fluang,  summarize  the 
work  of  some  30  papers  and  reports  that  deal  directly  with  prototyping  or  touch  on  the 
subject.  They  define  prototyping  as  “a  conscious  strategy  to  obtain  certain  types  of 
information  to  inform  specific  decisions”  (Birkler,  2010,  p.  64).  They  review  outcomes  from 
both  case  studies  and  statistical  analyses  of  prototyping  and  nonprototyping  programs. 

While  case  studies  provide  mixed  results,  they  conclude  that  statistical  analyses  show  that 
programs  applying  prototyping  knowledge  prior  to  the  development  decision  (today’s 
Milestone  B)  experience  less  cost  growth  (Birkler,  p.  76). 

Since  2002,  the  Government  Accountability  Office  (GAO)  has  undertaken  an  annual 
assessment  of  selected  defense  weapon  programs.  The  GAO  continues  to  advocate  a 
knowledge-based  acquisition  approach.  They  assess  the  knowledge  attained  by  each 
program  in  terms  of  requirements  matching  resources  and  maturity  of  technology  prior  to 
MS  B,  a  stable  design  at  the  Critical  Design  Review  (CDR),  and  capable  and  controlled 
production  processes  prior  to  MS  C.  In  the  past  two  years,  these  assessments  have 
included  a  useful  survey  of  selected  programs  in  order  to  determine  if  policies,  to  include  the 
policies  discussed  in  the  paper,  are  being  followed. 

There  are  two  limitations  inherent  in  the  GAO  assessments.  First,  by  design,  the 
assessments  only  address  selected  MDAP-level  defense  weapon  system  programs,  and 
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selected  potential  future  MDAPs,  not  all  programs.  The  2012  GAO  assessment  looked  at  37 
out  of  96  total  MDAPs  and  16  future  potential  MDAPs.  The  2013  GAO  assessment  looked  at 
40  out  of  86  total  MDAPs  and  17  future  potential  MDAPs.  Second,  the  assessments  are 
qualitative  vice  quantitative.  Only  limited  statistical  analyses  are  performed  on  the  selected 
programs  (GAO-1 2-400SP,  2012  and  GAO-1 3-294SP,  2013). 

There  are  numerous  reports  and  publications  with  anecdotal  references  to  the  value 
of  competitive  prototyping  to  identify  technology  maturity  and  the  importance  of  making 
requirements  trade-offs  at  the  preliminary  design  reviews.  One  such  reference  is  instructive. 
Thomas  Christie,  who  retired  in  2005  after  4  years  as  the  Director  of  Operational  Test  and 
Evaluation,  is  a  vocal  advocate  of  competitive  prototyping  and  the  other  reforms  in  the 
WSARA  of  2009.  In  his  essay  entitled,  Developing,  Buying  and  Fielding  Superior  Weapon 
Systems,  Christie  recommends  that  a  “Fly-before-Decide”  policy  of  competitive  prototyping 
be  mandated  for  all  programs  (not  just  MDAPs)  before  entering  the  EMD  phase  (Christie, 

2011,  pp.  121-122). 


Research  Methodology 

Cost  growth  percentages  for  all  MDAPs  were  obtained  from  Unit  Cost  Reports  (UCR) 
included  in  Selected  Acquisition  Reports  (SARs)  from  the  years  2011  and  2012.  All  MDAPs 
submit  an  annual  SAR,  to  include  an  UCR,  to  the  Congress.  To  handle  changes  in 
procurement  quantities  over  time,  a  unit  cost  number,  Program  Acquisition  Unit  Cost 
(PAUC),  was  used  to  determine  cost  growth.  PAUC  is  calculated  as  follows: 

PAUC  ~  RDT &E + Pro  cur  ement+ System  specific  Military  Constuction  costs  ^ 

Total  number  of  fully  configured  end  items  procured  ( to  include  R&D  units )  v  ' 


The  original  PAUC,  in  base-year  dollars,  was  taken  from  the  UCR)  submitted  with 
the  first  annual  SAR  after  MS  B  for  development  programs  and  MS  C  for  production 
programs.  In  either  case,  this  original  PAUC  was  determined  after  competitive  prototyping 
and  after  PDRs,  if  those  activities  were  part  of  the  program’s  acquisition  strategy.  This 
original  PAUC  was  compared  with  the  current  UCR  PAUC  estimate,  calculated  to  the  same 
base-year  dollars,  and  reported  in  the  latest  annual  SAR  (201 1  and  2012).  Thus,  the  cost 
growth  percentage  was  determined  as  follows: 


Percent  Change  to  Date  in  PAUC 


Current  PAUC-Original  PAUC 
Original  PAUC 


x  100  % 


(2) 


In  addition,  the  annual  SARs  identifies  if  programs  have  suffered  an  Acquisition 
Program  Baseline  (APB)  threshold  schedule  breach.  A  program  breaches  its  APB  threshold 
schedule  when  its  schedule  slips  beyond  the  number  of  months  agreed  to  in  the  APB  signed 
by  the  Program  Manager  and  the  MDA.  For  SAR  reporting  purposes,  a  schedule  breach  of  6 
months  or  more  from  the  baseline  results  in  a  reportable  breach. 

For  the  past  two  years,  as  part  of  its  annual  assessments  of  selected  weapon 
programs,  the  Government  Accountability  Office  (GAO)  has  conducted  a  survey  of  current 
and  future  Major  Defense  Acquisition  Programs  (MDAPs).  The  purpose  of  the  survey  is  to 
determine  if  various  acquisition  reform  policies  and  practices  are  being  implemented.  From 
the  survey  data,  the  GAO  has  been  able  to  identify  some  programs  that  have  demonstrated 
technology  maturity  on  prototypes  in  a  relevant  environment  (Technology  Readiness  Level 
6)  and  have  conducted  a  preliminary  design  review  prior  to  Milestone  B  (GAO-1 2-400SP, 
2012  and  GAO-1 3-294SP,  2013).  These  programs  are  listed  in  Appendix  B. 

Descriptive  statistics  are  used  to  analyze  cost  growth  (percent  change  to  date  in 
PAUC)  and  schedule  breaches  for  the  MDAPs  that  have  conducted  competitive  prototyping 
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and  PDR  activities.  Similar  descriptive  statistics  are  used  to  analyze  the  balance  of  the 
MDAPs  included  in  a  particular  annual  SAR  submission.  For  example,  there  were  89  total 
MDAPs  reported  in  the  201 1  SAR  submission.  Of  these  MDAPs,  33  were  identified  by  the 
GAO  survey  as  having  demonstrated  technology  maturity  on  prototypes  in  a  relevant 
environment  (TRL  6)  and  having  conducted  a  PDR.  The  remaining  56  programs  may  or  may 
not  have  used  prototypes  or  achieved  TRL  6  or  conducted  a  PDR.  The  percentage  of 
programs  that  have  negative  cost  growth  (negative  percent  change  to  date  in  PAUC)  from 
each  population  is  compared.  The  population  with  the  highest  number  of  negative  cost 
growth  programs  is  preferred.  Similarly,  the  percentage  of  programs  that  suffered  an  APB 
schedule  threshold  breach  from  each  population  is  compared.  The  population  with  the 
lowest  percent  of  schedule  breaches  is  preferred. 

Results  Analysis  and  Conclusion 

Research  results  are  summarized  in  Tables  2  through  5  of  Appendix  C. 

PAUC  Cost  Growth  Results.  Based  upon  data  from  the  201 1  SAR  (Table  2), 
programs  that  demonstrated  technology  maturity  on  prototypes  in  a  relevant  environment 
(TRL  6)  and  conducted  a  preliminary  design  review  prior  to  Milestone  B  were  more  often  to 
show  negative  PAUC  cost  growth.  Moreover,  this  result  was  seen  in  all  DoD  Components. 
Similar  results  were  found  in  the  data  from  the  2012  SAR  (Table  3). 

Caution  is  needed  when  interpreting  these  results.  While  all  cost  growth  was 
measured  from  the  original  baseline,  that  baseline  may  have  been  different,  depending  on 
the  point  at  which  the  program  was  initiated.  Most  programs  establish  their  original  baseline 
at  the  development  decision,  MS  B.  However,  programs  making  use  of  non-developmental 
and  commercial  off-the-shelf  items  may  be  approved  for  program  initiation  at  the  production 
decision,  MS  C.  Thus,  the  original  baseline  upon  which  cost  growth  is  measured  would  also 
be  established  at  MS  C,  vice  MS  B. 

Cost  growth  was  measured  based  upon  the  change  in  cost  estimates,  not  actual 
costs.  Cost  estimates  change  for  various  reasons.  The  annual  SAR  reports  (DoD,  201 1  and 
DoD,  2012)  show  these  categories  of  reasons  for  cost  changes  between  the  baseline  and 
current  cost  estimates:  quantity,  schedule,  engineering,  support,  estimating,  and  other.  By 
using  PAUC  as  the  cost  metric,  this  research  has  adjusted  for  changes  in  the  cost  estimates 
due  to  quantity.  Cost  changes  due  to  schedule,  engineering,  and  support  could  be  directly 
attributable  to  the  use  of  competitive  prototypes  to  identify  technology  maturity  and  the  use 
of  a  PDR  prior  to  MS  B.  However,  cost  changes  due  to  estimating  would  indicate  that 
competitive  prototyping  and  an  early  PDR  did  not  help  refine  the  cost  estimate. 

Schedule  Threshold  Breach  Results.  Based  upon  data  from  the  201 1  SAR  (Table  4), 
programs  that  demonstrated  technology  maturity  on  prototypes  in  a  relevant  environment 
(TRL  6)  and  conducted  a  preliminary  design  review  prior  to  Milestone  B  did  not  suffer  fewer 
APB  schedule  threshold  breaches.  Similar  results  were  found  in  the  data  from  the  2012  SAR 
(Table  5). 

While  the  cost  growth  results  seem  to  support  the  value  of  demonstrating  technology 
maturity  on  prototypes  (TRL  6)  and  conducting  an  early  preliminary  design  review  (before 
MS  B),  the  schedule  threshold  breach  results  do  not.  There  may  be  several  reasons  for  this. 

First,  a  reportable  schedule  breach  is  6  months  of  schedule  slippage  from  the 
previous  SAR.  All  MDAPS  must  submit  an  annual  SAR.  And,  if  a  unit  cost  or  schedule 
breach  occurs,  the  program  must  submit  a  SAR  every  quarter.  So,  a  schedule  breach  is 
going  to  be  reported  relatively  quickly  after  it  occurs.  In  this  research,  the  schedule  breach 
metric  only  indicates  that  there  was  a  schedule  slippage  of  at  least  6  months.  The  metric 
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does  not  quantify  the  total  number  of  months  of  slippage  suffered  by  the  program.  Thus,  if 
the  two  populations  of  programs  were  compared  using  the  total  months  of  schedule 
slippage,  there  might  have  been  a  different  outcome. 

Like  cost  estimates,  baseline  schedules  are  also  estimates  that  can  change  for 
reasons  other  than  execution  performance  of  the  development  and  production  efforts. 
Funding  availability  can  delay  a  development  or  disrupt  production.  For  example,  Warfighter 
Information  Network-Tactical  (WIN-T)  Increment  3  breached  its  threshold  schedule  due  to  a 
decrement  in  the  FY  2014  President’s  Budget  (DoD,  201 1 ).  Alignment  of  supporting 
program  schedules  can  also  cause  delays.  For  example,  a  3-year  slippage  in  the  Initial 
Operational  Capability  (IOC)  date  for  the  Littoral  Combat  Ship  (LCS)  Mine  Countermeasures 
Mission  (MCM)  package  caused  a  reportable  schedule  breach  for  the  MH-60S  Multi-Mission 
Helicopter  (DoD,  201 1 ). 

Finally,  a  comparison  should  be  made  between  the  earlier  research  by  RAND  based 
upon  1960-1980  program  data  (Drezner,  1992)  and  this  research  based  upon  1990-2012 
program  data.  Drezner’s  work  was  done  in  an  era  when  defense  acquisition  programs  were 
not  required  to  use  TRLs  to  assess  the  maturity  of  critical  technologies.  The  DoD  adopted 
TRLs  from  the  National  Aeronautics  and  Space  Administration  (NASA)  and  first  introduced 
them  as  a  tool  for  program  managers  in  the  early  2000s.  In  addition,  prior  to  the  2008 
version  of  DoD  Instruction  5000.2,  acquisition  policy  did  not  emphasize  the  importance  of 
the  PDR.  DoD  Instruction  5000.02  (2008)  elevated  oversight  of  the  PDR  by  requiring  a  Post 
Preliminary  Design  Review  Assessment  (Post-PDRA)  at  the  MDA  level  (DoD,  2008).  This 
emphasis  is  certain  to  have  caught  the  attention  of  program  managers  and  motivated  them 
to  pay  more  attention  to  technology  readiness  and  PDR  policies.  Perhaps  this  recent 
emphasis  has  had  an  effect  on  reducing  cost  growth  and  schedule  slippages  in  the 
programs  used  in  this  research. 

Recommendations  for  Further  Research 

It  is  doubtful  that  the  USD[AT&L]  will  ever  be  able  to  truly  determine  the  effect  of  the 
competitive  prototyping,  technology  readiness,  and  early  PDR  policies  on  program  cost 
growth  or  schedule  slippage.  As  Drezner  said,  “One  fundamental  problem  is  that  while  we 
can  examine  prototyping  programs,  or  compare  the  outcomes  of  prototyping  and 
nonprototyping  programs,  the  outcome  for  the  same  program  with  and  without  prototyping 
can  never  be  known”  (Drezner,  1992,  p.  59).  However,  there  are  some  things  that  can  be 
done  to  improve  the  imperfect  research  to  date. 

First,  to  determine  cost  growth,  compare  PAUC  based  upon  the  original  cost 
estimate  with  actual  PAUC.  Actual  PAUC  can  be  determined  from  contracts  found  in  the 
Defense  Cost  and  Resource  Center  (DCARC)  database.  The  percentage  of  growth  from  the 
PAUC  based  upon  the  original  cost  estimate  (at  program  initiation)  and  the  actual  PAUC  (at 
the  end  of  the  development  and  production  program)  would  remove  some  of  the  uncertainty 
in  this  metric. 

Second,  to  determine  schedule  slippage,  compare  the  original  schedule  estimate 
with  actual  schedule  performance  data.  Again,  the  actual  schedule  performance  data  for  this 
comparison  should  be  available  in  the  DCARC  database  or  Defense  Acquisition 
Management  Information  Retrieval  (DAMIR).  The  percentage  of  growth  from  the  original 
schedule  estimate  to  the  actual  schedule  performance  would  remove  some  of  the 
uncertainty  in  this  metric. 

Finally,  the  challenge  in  using  cost  growth  and  schedule  slippage  metrics  is  to  tie 
them  back  to  the  use  of  competitive  prototyping  (to  reveal  technology  readiness)  and  the 
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use  of  an  early  PDR.  Drezner  said  that  prototyping  is  “a  conscious  strategy  to  obtain  certain 
types  of  information  to  inform  specific  decisions”  (Birkler,  2010,  p.  64).  The  same  can  be 
said  for  technology  readiness  and  PDRs.  The  knowledge  from  these  activities  and  how  that 
knowledge  is  applied  will  tell  us  whether  these  policies  have  had  an  effect.  To  that  end, 
more  detailed  surveys,  such  as  those  conducted  annually  on  selected  weapon  systems  by 
the  GAO,  will  aid  in  helping  establish  the  cause-effect  relationship  between  policy  and 
program  outcomes. 
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Appendix  A.  Technology  Readiness  Level  (TRL)  Descriptions 

(Defense  Acquisition  Guidebook,  2013,  para.  10.5.2.2) 


Technology  Readiness  Level 

Description 

1.  Basic  principles  observed  and 
reported. 

Lowest  level  of  technology  readiness.  Scientific  research  begins  to  be 
translated  into  applied  research  and  development.  Examples  might 
include  paper  studies  of  a  technology's  basic  properties. 

2.  Technology  concept  and/or 
application  formulated. 

Invention  begins.  Once  basic  principles  are  observed,  practical 
applications  can  be  invented.  Applications  are  speculative  and  there 
may  be  no  proof  or  detailed  analysis  to  support  the  assumptions. 
Examples  are  limited  to  analytic  studies 

3  Analytical  and  experimental 
critical  function  and/or 
characteristic  proof  of  concept. 

Active  research  and  development  is  initiated.  This  includes  analytical 
studies  and  laboratory  studies  to  physically  validate  analytical 
predictions  of  separate  elements  of  the  technology.  Examples  include 
components  that  are  not  yet  integrated  or  representative. 

4  Component  and/or 
breadboard  validation  in 
laboratory  environment. 

Basic  technological  components  are  integrated  to  establish  that  they 
will  work  together.  This  is  relatively  "low  fidelity"  compared  to  the 
eventual  system.  Examples  include  integration  of  "ad  hoc”  hardware 
in  the  laboratory. 

5.  Component  and/or 
breadboard  validation  in  relevant 
environment. 

Fidelity  of  breadboard  technology  increases  significantly.  The  basic 
technological  components  are  integrated  with  reasonably  realistic 
supporting  elements  so  it  can  be  tested  in  a  simulated  environment. 
Examples  include  "high  fidelity"  laboratory  integration  of  components. 

6.  System/subsystem  model  or 
prototype  demonstration  in  a 
relevant  environment. 

Representative  model  or  prototype  system,  which  is  well  beyond  that 
of  TRL  5:  is  tested  in  a  relevant  environment.  Represents  a  major 
step  up  in  a  technology's  demonstrated  readiness.  Examples  include 
testing  a  prototype  in  a  high-fidelity  laboratory  environment  or  in 
simulated  operational  environment, 

7.  System  prototype 
demonstration  in  an  operational 
environment 

Prototype  near,  or  at,  planned  operational  system  Represents  a 
major  step  up  from  TRL  6,  requiring  demonstration  of  an  actual 
system  prototype  in  an  operational  environment  such  as  an  aircraft, 
vehicle,  or  space.  Examples  include  testing  the  prototype  in  a  test 
bed  aircraft. 

3  Actual  system  completed  and 
qualified  through  test  and 
demonstration. 

Technology  has  been  proven  to  work  in  its  final  form  and  under 
expected  conditions.  In  almost  all  cases,  this  TRL  represents  the  end 
of  true  system  development.  Examples  include  developmental  test 
and  evaluation  of  the  system  in  its  intended  weapon  system  to 
determine  if  it  meets  design  specifications. 

9.  Actual  system  proven  through 
successful  mission  operations. 

Actual  application  of  the  technology  in  its  final  form  and  under  mission 
conditions,  such  as  those  encountered  in  operational  test  and 
evaluation.  Examples  include  using  the  system  under  operational 
mission  conditions. 
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Appendix  B.  Programs  Identified  as  Using  Competitive  Prototyping  and 
Preliminary  Design  Reviews  (PDR) 


GAO-1 2-400 SP  (33  Programs) 

Component 

AGM-88E  Advanced  Anti-Radiation  Guided  Missile  (AARGM) 

USN 

Apache  Block  IIIA  (ABBA)  Remanufacture 

USA 

Airborne  and  Maritime/Fixed  Station  Joint  Tactical  Radio  System  (AMF  JTRS) 

DoD 

B-2  Extremely  High  Frequency  SATCOM  Capability,  Increment  1 

USAF 

C-130  Avionics  Modernization  Program  (AMP) 

USAF 

CH-53K  Heavy  Lift  Replacement  (CH-53K) 

USN 

DDG  1000  Zumwalt  Class  Destroyer 

USN 

E-2D  Advanced  Hawkeye  (AHE) 

USN 

Excalibur  Precision  155mm  Projectiles 

USA 

Family  of  Advanced  Beyond  Line-of -Sight  Terminals  (FAB-T) 

USAF 

Global  Positioning  System  (GPS)  111 

USAF 

HC/MC-130  Recapitalization  Program 

USAF 

Integrated  Air  &  Missile  Defense  (IAMD) 

USA 

Integrated  Defensive  Electronic  Countermeasures  (IDECM)  Block  4 

USN 

Joint  Air-to-Surface  Standoff  Missile  (JASSM  Baseline) 

USAF 

Joint  Air-to-Surface  Standoff  Missile  (JASSM-Extended  Range) 

USAF 

Joint  High  Speed  Vessel  (JHSV) 

USN 

Joint  Land  Attack  Cruise  Missile  Defense  Elevated  Netted  Sensor  System  (JLENS) 

USA 

Joint  Precision  Approach  and  Landing  System  (JPALS) 

USN 

LHA  6  America  Class  Amphibious  Assault  Ship 

USN 

Littoral  Combat  Ship  (LCS) — Seaframes 

USN 

Mobile  User  Objective  System  (MUOS) 

USN 

MQ-1C  Unmanned  Aircraft  System  (UAS)  Gray  Eagle 

USA 

MQ-4C  Triton  Unmanned  Aircraft  System  (MQ-4C  Triton) 

USN 

MQ-9  Unmanned  Aircraft  System  (UAS)  Reaper 

USAF 

Navy  Multiband  Terminal  (NMT) 

USN 

P-8A  Poseidon 

USN 

RQ-4A/B  Global  Hawk  Unmanned  Aircraft  System 

USAF 

Small  Diameter  Bomb  Increment  II  (SDB  11} 

USAF 

Standard  Missile-6  (SM-6) 

USN 

Vertical  Takeoff  and  Landing  Tactical  Unmanned  Aerial  Vehicle  -  MQ-8  Fire  Scout 
(VTUAV) 

USN 

Warfighter  Information  Network-Tactical  (WIN-T)  Increment  2 

USA 

Warfighter  Information  Network-Tactical  (WIN-T)  Increment  3 

USA 
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GAO-13-294SP  (34  Programs) 

Component 

AH-64E  Apache  Remanufacture 

USA 

AIM-9X  Block  11  Air-to-Air  Missile 

USN 

AN/TPS-80  Ground/Air  Task  Oriented  Radar  (G/ATOR) 

USN 

CH-53K  Heavy  Lift  Replacement  (CH-53K) 

USN 

DDG  1000  Zumwalt  Class  Destroyer 

USN 

E-2D  Advanced  Hawkeye  (AHE) 

USN 

Excalibur  Precision  155mm  Projectiles 

USA 

Family  of  Advanced  Beyond  Line-of-Sight  Terminals  (FAB-T) 

USAF 

Gerald  R.  Ford  Class  Nuclear  Aircraft  Carrier  (CVN  78  Class) 

USN 

Global  Positioning  System  (GPS)  111 

USAF 

HC/MC-130  Recapitalization  Program 

USAF 

Integrated  Air  &  Missile  Defense  ((AMD) 

USA 

Integrated  Defensive  Electronic  Countermeasures  (IDECM)  Block  4 

USN 

Joint  Air-to-Surface  Standoff  Missile  (JASSM  Baseline) 

USAF 

Joint  Air-to-Surface  Standoff  Missile  (JASSM-Extended  Range) 

USAF 

Joint  High  Speed  Vessel  (JHSV) 

USN 

Joint  Precision  Approach  and  Landing  System  (JPALS) 

USN 

Joint  Tactical  Radio  System  (JTRS)  Handheld,  Manpack,  and  (HMS)  Small  Form  Fit 
Radios 

USA 

KC-46A  Tanker  Modernization  Program 

USAF 

LHA  6  America  Class  Amphibious  Assault  Ship 

USN 

Littoral  Combat  Ship  (LCS)— Seaframes 

USN 

Mobile  User  Objective  System  (MUOS) 

USN 

MQ-1C  Unmanned  Aircraft  System  (UAS)  Gray  Eagle 

USA 

MQ-4C  Triton  Unmanned  Aircraft  System  (MQ-4C  Triton) 

USN 

MQ-9  Unmanned  Aircraft  System  (UAS)  Reaper 

USAF 

Navy  Multi  band  Terminal  (NMT) 

USN 

P-8  A  Poseidon 

USN 

Paladin/Field  Artillery  Ammunition  Support  Vehicle  (FAASV)  Integrated 

Management  (PlM) 

USA 

RQ-4A/B  Global  Hawk  Unmanned  Aircraft  System 

USAF 

Ship  to  Shore  Connector  (SSC) 

USN 

Standard  Missile-6  (SM-6) 

USN 

Vertical  Takeoff  and  Landing  Tactical  Unmanned  Aerial  Vehicle  -  MQ-8  Fire  Scout 
(VTUAV) 

USN 

Warfighter  Information  Network-Tactical  (WIN-T)  Increment  2 

USA 

Warfighter  Information  Network-Tactical  (WIN-T)  Increment  3 

USA 
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Appendix  C.  Results  of  Research 

Table  2.  Programs  Costing  Less,  Selected  Acquisition  Report,  December  31,  2011 


Component 

Programs  w/Prototypes  &  PDR 

Balance  of  Programs 

Programs 
Costing  Less 

Total 

Programs 

Percent 

Programs 

Costing 

Less 

Total 

Programs 

Percent 

Army 

6 

7 

86 

3 

~1T 

25 

Navy 

7 

15 

47 

6 

— W 

30 

Air  Force 

5 

50 

4 

~w 

27 

Def  Agency 

1 

i 

100 

2 

~1T 

Total 

19 

33 

57 

15 

56 

27 

Table  3.  Programs  Costing  Less,  Selected  Acquisition  Report,  December  31,  2012 


Component 

Programs  w/Prototypes  &  PDR 

Balance  of  Programs 

Programs 
Costing  Less 

Total 

Programs 

Percent 

Programs 

Costing 

Less 

Total 

Programs 

Percent 

Army 

“5" 

8 

62 

4 

~\T 

“33" 

Navy 

8 

18 

44~ 

4 

20 

“20" 

Air  Force 

3 

8 

38 

6 

17 

35 

Def  Agency 

0 

0 

0 

4 

5 

80 

Total 

16 

34 

47 

18 

54 

33 

Table  4.  Program  Schedule  Breach,  Selected  Acquisition  Report,  December  31,  2011 


Component 

Programs  w/Prototypes  &  PDR 

Balance  of  Programs 

Programs 

w/Schedule 

Breach 

Total 

Programs 

Percent 

Programs 

w/Schedule 

Breach 

Total 

Programs 

Percent 

Army 

2 

~T 

28 

2 

12 

17 

Navy 

4 

15 

27 

5 

20 

25 

Air  Force 

4 

10 

40 

6 

15 

40 

Def  Agency 

1 

1 

100 

4 

9 

44 

Total 

11 

33 

33 

17 

56 

30 

Table  5.  Program  Schedule  Breach,  Selected  Acquisition  Report,  December  31,  2012 


Component 

Programs  w/Prototypes  &  PDR 

Balance  of  Programs 

Programs 

w/Schedule 

Breach 

Total 

Programs 

Percent 

Programs 

w/Schedule 

Breach 

Total 

Programs 

Percent 

Army 

3 

8 

38 

4 

12 

33 

Navy 

6 

18 

33 

3 

20 

15 

Air  Force 

~T 

8 

25 

7 

~TT\ 

41 

Def  Agency 

0 

0 

0 

0 

5 

0 

Total 

34 

30 

14 

54 

26 
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